Image processing apparatus and image processing method

ABSTRACT

An image processing apparatus is configured to reduce a color blur in a color image, and includes a color blur process determination part configured to change at least one of a color blur that serves as a reduction object and an estimation method of an amount of the color blur according to at least one of a characteristic of an image and an imaging condition.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing apparatus and animage processing method which are configured to provide a so-calledcolor blur reduction process to a color image obtained from the imaging.

2. Description of the Related Art

A color image pickup system generates an originally nonexistent color asa color blur around a bright part on an image due to a chromaticaberration of an imaging optical system. The color blur is likely tooccur at part distant from a central wavelength of the imaging opticalsystem, and a visible-light color image pickup system causes artifactsin blue, in red, in purple that is a blend of blue and red in a blurshape. This color blur is also referred to as a purple fringe.

A combination of plural lenses having different dispersions canoptically reduce the chromatic aberration to some extent. However, as asmall size of a digital camera progresses, high resolution image sensors(or image pickup devices) and smaller optical systems are increasinglydemanded, and it is difficult to sufficiently reduce the chromaticaberration only by the optics. Therefore, the image processing isrequired to reduce an artifact.

The chromatic aberration is roughly classified into a lateral chromaticaberration (chromatic aberration of magnification) and a longitudinal(or axial) chromatic aberration. When the light from a light sourceimages on a focal plane via an imaging optical system, as shown in FIG.1, the lateral chromatic aberration appears as shifts of imagingpositions of a B(Blue) ray having a wavelength of about 450 nm, aG(Green) ray having a wavelength of about 550 nm, and a R(Red) rayhaving a wavelength of about 650 nm shift in the image surfacedirection. On the other hand, when the light from a light source imageson a focal plane via an imaging optical system, as shown in FIG. 2, thelongitudinal chromatic aberration appears as shifts of imaging positionsof the RBG rays in the optical-axis direction.

As disclosed in U.S. Pat. No. 6,724,702, the lateral chromaticconversion of a digital imaging system of a primary color system can becorrected through a geometric conversion that applies a differentdeformation to each color plane of RGB.

On the other hand, the longitudinal chromatic aberration means, in animage that is focused with the G (Green) plane as a central wavelengthof the visible light region, for example, that the image is out of focuswith respect to the R(Red) plane and B(Blue) plane that are ends of thevisible light. This cannot be corrected by the geometric conversionalthough it is effective to the lateral chromatic aberration. Thelongitudinal chromatic aberration can be made inconspicuous by loweringthe chroma of a color blur area as disclosed in Japanese PatentLaid-Open No. (“JP”) 2001-145117, or can be corrected by applying adifferent contour emphasis process to each color plane of RGB asdisclosed in JP 2003-018407.

However, a chroma lowering process disclosed in JP 2001-145117 is aprocess that degrades a color of a bright pixel down to a brightnesslevel of another color. This process is effective in reducingunnaturalness by erasing a color of the color blur, but the originalcolors of a subject can sometimes be also influenced and the image islikely grayish irrespective of the existence of the color blur.

The process disclosed in JP 2003-018407 utilizes deconvolution and itsapproximate contour emphasis process, and is performed by calculating afilter parameter. However, this process fixes G as a reference, andcannot handle a wide variety of color blurs.

The conventional color blur reduction process always provides the sameprocess to blurs that can differ according to images, and cause anunnatural color depending upon a type of a blur or cannot sufficientlyreduce or leave a color blur.

SUMMARY OF THE INVENTION

The present invention is directed to an image processing apparatus andan image processing method which can properly change a color blurreduction process to a wide variety of color blurs in color imagesobtained through the photography, effectively reduce a color blur, andreproduce an original color of a subject.

An image processing apparatus according to one aspect of the presentinvention is configured to perform a process to reduce a color blur in acolor image. The image processing apparatus includes a color blurprocess determination part that changes at least one of a color blurthat serves as a reduction object and an estimation method of an amountof the color blur according to at least one of a characteristic of animage and an imaging condition.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments (with reference to theattached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an explanatory view of the lateral chromatic aberration.

FIG. 2 is an explanatory view of the longitudinal chromatic aberration.

FIG. 3 is a block diagram showing a structure of a color image pickupapparatus according to a first embodiment of the present invention.

FIG. 4 is an explanatory view of a primary color filter.

FIG. 5 is an explanatory view of a color filter array.

FIG. 6 is an explanatory view of a color blur at the boundary betweenthe brightness and the darkness.

FIG. 7 is a flowchart of a structure (processing procedure) of a colorblur process determination part in the image pickup apparatus accordingto the first embodiment.

FIG. 8 is a view for explaining a color blur when a brightnessdifference is large.

FIG. 9 is a block diagram of a structure of a color image pickupapparatus according to a second embodiment according to the presentinvention.

FIG. 10 is a flowchart of a structure (processing procedure) of a colorblur process determination part in the image pickup apparatus accordingto the second embodiment.

FIG. 11 is a view for explaining values of RGB planes when a bluishimage has a blue blur in an area having a large brightness difference.

FIG. 12 is a view for explaining values of the RGB planes when a bluishimage has a red blur in an area having a large brightness difference.

FIG. 13 is a block diagram of a structure of a color image pickupapparatus according to a third embodiment of the present invention.

FIG. 14 a flowchart of a structure (processing procedure) of a colorblur process determination part in the image pickup apparatus accordingto the third embodiment.

FIG. 15 is a view for explaining values of the RGB planes when a reddishimage has no color blur in an area having a large brightness difference.

FIG. 16 is a view for explaining values of the RGB planes when a reddishimage has a red blur in an area having a large brightness difference.

FIG. 17 is a view for explaining values of the RGB planes when a reddishimage has a blue blur in an area having a large brightness difference.

FIG. 18 is a view for explaining values of the RGB planes when the Rplane is in the on-focus state in an area having a large brightnessdifference of a reddish image.

FIG. 19 is a view for explaining a distance from a brightness saturationarea and a distance from a brightness unsaturation area.

FIG. 20 is a block diagram of a structure of a color image pickupapparatus according to a fourth embodiment of the present invention.

FIG. 21 a flowchart of a structure (processing procedure) of a colorblur process determination part in the image pickup apparatus accordingto the fourth embodiment.

DESCRIPTION OF THE EMBODIMENTS

Referring now to the drawings, a description will be given of apreferred embodiment of the present invention.

First Embodiment

FIG. 3 shows a color image pickup apparatus 100 that utilizes an imageprocessing method or is mounted with an image processing apparatusaccording to a first embodiment of the present invention.

The color image pickup apparatus 100 includes an imaging optical system110, an image sensor 120, such as a CCD sensor and a CMOS sensor, an ADconverter 130, a demosaic part 140, a color blur process determinationpart 150, a color blur reducer 160, a sight corrector 170, a compressionpart 180, and a recorder 190.

Although a field shown in FIG. 3 (imaging area or subject) f and therays of R (red), G (green), and B (blue) are not the elements of thecolor image pickup apparatus 100, they are illustrated for descriptionpurposes.

In FIG. 3, the taken field f is imaged on the image sensor 120 via theimaging optical system 110. In general, an imaging optical system in acolor image pickup apparatus is provided with a certain chromaticaberration correction. The longitudinal chromatic aberration of theimaging optical system 110 of this embodiment is corrected for allwavelength regions to some extent, and well corrected particularly forthe G wavelength region; a more longitudinal chromatic aberration leavesfor the other wavelength regions than that for the G wavelength region.Thus, other aberration corrections can be improved and the image pickupapparatus can be made smaller by degrading the correction criteria ofthe longitudinal chromatic aberration in the B wavelength region.

The image sensor 120 is a single-plate color image sensor having ageneral primary color filter system. The primary color filter systemincludes three types of color filters having main transmission bandsnear 650 nm, 550 nm, and 450 nm, as shown in FIG. 4, and images colorplanes corresponding to the RGB bands. The single-plate color imagesensor spatially arranges these color filters for each pixel, as shownin FIG. 5, and each pixel can obtain the intensity of only a singlecolor plane. Therefore, a color mosaic image is output from the imagesensor 120.

A three-plate type color image sensor may be used, which splits theincident light into the RGB wavelength regions using a color splittingprism (not shown) and images the light having a different wavelengthwith a separate image sensor. In this case, the demosaic part 140 isunnecessary.

The AD converter 130 converts a color mosaic image output as an analogvoltage from the image sensor 120, into digital data suitable for thesubsequent image processing.

The demosaic part 140 interpolates a color mosaic image, and generates acolor image in which all pixels have RGB color information. A variety ofmethods are proposed for the interpolation approach from a simple linearinterpolation to a complicated approach introduced in E. Chang, S.Cheung, and D. Pan, “Color filter array recovery using a threshold-basedvariable number of gradients,” Proc. SPIE, vol. 3650, pp. 36-43, January1999. The interpolation approach of this embodiment can use any one ofthese methods or another method.

The color image generated by the demosaic part 140 has resolutions ofthe R (red) plane and the B (blue) plane lower than the resolution ofthe G (green) plane due to the chromatic aberration of the imagingoptical system 110. Therefore, color blurs or fuzzes of R and B occur atthe boundary between the brightness and the darkness, for example, asshown in FIG. 6, and there occur fringes or artifacts of R, B and purpleas a blend of R and B around the bright part. The blur levels aredifferent between R and B due to the characteristic of the imagingoptical system 110 and the spectroscopic distribution of the lightsource in the image.

This embodiment uses a RGB primary color system for a color filter ofthe image sensor 120, but may use a color filter of a complementarycolor system, which provides a color image composed of the RGB colorplanes through the color conversion process.

The color blur process determination part 150 determines the processcontent to be performed by the color blur reducer 160, according to acharacteristic (or “feature” hereinafter) of an input color image and animaging condition, as detailed later.

The color blur reducer 160 extracts a color blur area through a spatialoperation, estimates a color blur amount, subtracts the estimated amountfrom the input color image, and reduces the color blur.

Next follows a process by the sight corrector 170. The sight corrector170 mainly performs a process that intends to improve an appearance of acolor image. This image correction includes, for example, a tone curve(gamma) correction, a chroma emphasis, a hue correction, and an edgeemphasis.

As the last process, the compression part 180 compresses the correctedimage using a method, such as JPEG, and reduces a size used forrecording.

The color image that has undergone the compression process is recordedby the recorder 190 in a recording medium, such as a hard disc drive, aDVD, a magnetic tape, and a flash memory.

Those processing components from the image sensor 120 to the recorder190 may actually include separate devices or a single microprocessor.

Referring now to the flowchart shown in FIG. 7, a description will begiven of a process or an image processing method by the color blurprocess determination part 150. The process at the color blur processdetermination part 150 includes an image feature calculation step S151,a color blur reduction object plane selection step S152, and anestimation parameter determination step S155. These processes areexecuted according to a computer program (image processing program)stored in a memory in an image pickup apparatus 100. This is true ofother embodiments, which will be described later.

A normal imaging optical system is designed with the G wavelength regionas a reference. Therefore, although it depends upon the imaging opticalsystem, a chromatic aberration from the R wavelength to the B wavelengthbasically appears as a color blur. Accordingly, this embodiment performsa process by fixing a color plane to be referred to as a reference tothe G plane

The image feature calculation step S151 calculates a brightness value ofthe entire image as a feature of the image relating to the color blur.It is supposed that an image in which the entire image has a lowbrightness value is highly likely to be an image of a night view or animage taken in a dark room. It is conceivable that an image of a nightview or an image taken in a dark room has an area having a highbrightness value in an area having a low brightness value.

In that case, a brightness difference increases, as shown in FIG. 8, anda color blur is highly likely to occur around the high brightness area.Since the on-focus state is set with the G wavelength, the color blursof R and B are likely to occur. In addition, as a brightness differenceincreases, a color blur radius from a brightness saturation areaincreases. Moreover, when a part having a low brightness value isblurred with a specific color plane component, a color blurconspicuously has a high chroma.

Accordingly, a parameter indicative of the intensity used to estimate acolor blur amount is made larger for a (first) image obtained fromimaging of the night view or in the dark room in which the entire imagehas a low brightness value than a (second) image in which the entirescreen has a high brightness value. This configuration can effectivelyreduce the color blur.

On the contrary, an image in which the entire screen has a highbrightness value is supposed to be an image obtained by the outsideimaging under the daylight. For this image, a color blur, if any, isless likely conspicuous because its brightness is wholly high, and anarea having a large brightness difference is less likely to occur.Therefore, a color blur plane as a reduction object is set to a defaultcolor plane (for example, B plane), and an intensity parameter is set sothat a color blur area can decrease and the estimated color blurintensity can be low. This configuration can estimate a small color bluramount, and avoid an excessive reduction process due to the color blurreduction process.

By calculating a brightness value of the entire image as describedabove, the color plane as the color blur reduction object can beselected and the spread of the color blur can be properly changed inestimating the color blur. In addition, a parameter indicative of theintensity used to estimate the color blur amount can be properlychanged.

Next, the color blur reduction object plane selection step S152determines a color blur reduction object plane or a reduction objectbased on the brightness value of the image obtained by the image featurecalculation step S151. For example, when an input image is determined tobe a night view image from a brightness value, both the R and B planesare selected as the color blur reduction object planes.

Next, the estimation parameter determination step S155 determines aparameter used for an estimation relating to the color blur, based onthe brightness value as the feature of the image obtained in the imagefeature calculation step S151. For example, when the brightness value ofthe entire screen is low, a parameter is set so that a color blur areacan widen and a color blur amount having a high chroma can be estimated.

One estimation method of a color blur amount executed at the color blurreducer 160 determines a color blur by a brightness slope betweenadjacent pixels. This method calculates a brightness slope map,estimates a color blur area using the brightness slope, and estimates acolor blur amount based on the brightness slope.

For example, when ∇R is a brightness slope map to the R plane and ∇B isa brightness slope map to the B plane, ∇R and ∇B are calculated asfollows:

$\begin{matrix}\begin{matrix}{{\nabla R} = \left( {\frac{\mathbb{d}R}{\mathbb{d}x},\frac{\mathbb{d}R}{\mathbb{d}y}} \right)} \\{\equiv \left( {\frac{{R\left( {{x + 1},y} \right)} - {R\left( {{x - 1},y} \right)}}{2},\frac{{R\left( {x,{y + 1}} \right)} - {R\left( {x,{y - 1}} \right)}}{2}} \right)}\end{matrix} & {{EQUATION}\mspace{14mu} 1} \\\begin{matrix}{{\nabla B} = \left( {\frac{\mathbb{d}B}{\mathbb{d}x},\frac{\mathbb{d}B}{\mathbb{d}y}} \right)} \\{\equiv \left( {\frac{{B\left( {{x + 1},y} \right)} - {B\left( {{x - 1},y} \right)}}{2},\frac{{B\left( {x,{y + 1}} \right)} - {B\left( {x,{y - 1}} \right)}}{2}} \right)}\end{matrix} & \;\end{matrix}$

R(x+1, y) and B(x+1, y) are values of the R and B planes for a pixelnext to the target pixel at the right side.

R(x−1, y) and B(x−1, y) are values of the R and B planes for a pixelnext to the target pixel at the left side.

R(x, y+1) and B(x, y+1) are values of the R and B planes for a pixelnext to the target pixel at the bottom side.

R(x, y−1) and B(x, y−1) are values of the R and B planes for a pixelnext to the target pixel at the top side.

A color blur area is determined based on a brightness slope value; whenthe brightness slope value is equal to or greater than a certain value,a color blur is determined. In that case, in addition to the brightnessslopes of R and B, a brightness slope of G as a reference plane iscalculated and compared to them so as to precisely identify the colorblur area.

Moreover, a color blur area can be more effectively identified bychanging a threshold of the brightness slope.

The estimation method of the color blur amount calculates estimatedcolor blur amounts E_(R) and E_(B) by multiplying absolute values of ∇Ras a brightness slope of R and ∇B as a brightness slope of B byparameters k_(R) and k_(B). In other words, the following equations areused to calculate the estimated color blur amounts E_(R) and E_(B) andthe process to reduce the estimated color blur amount follows.E _(R) =k _(R) |∇R|E _(B) =k _(B) |∇B|  EQUATION 2

With such a color blur reduction approach, the color blur estimationparameter determination step S155 determines values of the parametersk_(R) and k_(B) based on the brightness value as the feature of theimage. For example, when the brightness value of the entire screen islow, the values of the parameters k_(R) and k_(B) are determined so asto widen the color blur area and to estimate a color blur amount havinga high chroma.

The image feature calculation step S151 of this embodiment calculatesthe brightness value of the entire image as the feature of the image,but may calculate the contrast of the entire image. An image in whichthe entire image has a high contrast is considered to have a largebrightness difference. A color blur is likely to occur at part having alarge brightness difference, a color blur area is wide, and a color blurhaving a high chroma is likely conspicuous. On the contrary, an image inwhich the entire image has a low contrast is considered to have a smallbrightness difference, and a color blur is less likely to occur, andless likely conspicuous even if it occurs.

Therefore, it is possible to set a large color blur estimation amount toa (first) image in which the entire image has a high contrast, and asmall color blur estimation amount to a (second) image in which theentire image has a low contrast.

Second Embodiment

Next follows a description of a second embodiment according to thepresent invention. The color image pickup apparatus 100′ of the secondembodiment includes, as shown in FIG. 9, an imaging optical system 110,an image sensor 120, an AD converter 130, a demosaic part 140, a colorblur process determination part 150′, a color blur reducer 160, a sightcorrector 170, a compression part 180, and a recorder 190. Thoseelements which are the same as corresponding elements in the firstembodiment are designated by the same reference numerals. Thisembodiment is different from the first embodiment in a process of thecolor blur process determination part 150′.

FIG. 10 shows the color blur process determination part 150′ of thisembodiment. The color blur process determination part 150′ has a colorblur reduction object plane selection step S152, and an estimationparameter determination step S155.

An imaging condition as well as an input color image is input into thecolor blur process determination part 150′. The color blur reductionobject plane selection step S152 determines a color blur reductionobject plane according to the information of the imaging condition.

One of the imaging conditions available to a determination of a colorblur reduction object plane is a setting of an imaging scene (or inother words, a setting of an imaging mode). When a photographer takes animage after intentionally setting a scene suitable for the imagingcondition, an image obtained by imaging is an image in which a scene hasbeen set. Therefore, a feature of the image can be obtained to someextent without calculating a feature (or a characteristic) of the image.

For example, when the scene setting is a night view (a night viewimaging scene or a night view imaging mode), the image is supposed to bean image obtained from the night view imaging. In case of a night viewimage, a background is dark and a light source, such as a streetlight,is likely to cause brightness saturation and a color blur is likely tooccur. Moreover, since the background is dark, a color blur having ahigh chroma appears in a wide range when a specific color planecomponent is blurred, and a color blur is likely conspicuous. Inaddition, the night view is in the on-focus state with the G plane, andsignificant amounts of the chromatic aberrations of R and B which cannotbe corrected and slightly remain appear around a brightness saturationarea by a high brightness light source. Therefore, color blurs of both Rand B planes are likely to stand out.

With the foregoing, the color blur reduction object plane selection stepS152 determines that it is desirable to set a color plane of thereduction object to both the R plane and the B plane for the scenesetting of the night view.

The estimation parameter determination step S155 changes a setting of aparameter that designates the spread of a color blur for a night viewimaging scene image (first image) so as to determine more part as acolor blur area than that for an imaging scene image (second image)other than the night view imaging scene. In addition, the estimationparameter determination step S155 sets a parameter used to estimate acolor blur amount so that a larger color blur amount can be estimated.

One estimation method of the color blur amount performed by the colorblur reducer 160 is to calculate a brightness saturation area and otherareas in an image, to perform a convolution process, and to determine acolor blur area and a color blur amount. An initial area divisiondivides a reference color plane into a saturation area and anunsaturation area by utilizing a certain threshold, and calculates asaturation degree S in which each area is expressed by a binary value.Subsequently, a convolution process is performed for this saturationdegree S so as to determine a color blur area and a color blur amount.When the color blur reduction object plane selection step S152 selectsthe R and B planes as the color blur reduction object plane, a colorblur amount of each plane is calculated through the followingconvolution processes, where S_(R) and S_(B) are color blurs of theseplanes:S _(R) =k _(R)

SS _(B) =k _(B)

S  EQUATION 3

k_(R) and k_(B) are convolution Kernals; as the Kernal sizes of k_(R)and k_(B) increase, a color blur area widens and a color blur intensityetc. can be set by a value of the Kernal.

Hence, when the color blur reducer 160 uses such a color blur reductionapproach, the color blur estimation parameter determination step S155determines a size and value of the convolution Kernal as a parameterthat designates a color blur area and a color blur intensity inaccordance with the scene setting.

Although the night view is used as a scene setting example, anotherscene setting (or another imaging mode) may be used to determine a colorblur. For example, when the scene setting is underwater, an imageobtained by imaging is estimated to be a bluish image. In case of abluish image, the entire image is bluish, as shown in FIG. 11, and thereare no sudden color changes and a few blue blurs are less likelyconspicuous. The problem arises when there is a reddish and highbrightness area in a bluish image. When there is a reddish and highbrightness area, a red blur is likely to occur around the reddish andhigh brightness area, as shown in FIG. 12. When FIGS. 11 and 12 arecompared with each other, a color different from the hue of the entireimage occurs and is likely conspicuous around the reddish and highbrightness area because the red is blurred in FIG. 12. Therefore, thecolor blur reduction object plane selection step S152 sets the colorblur reduction object plane to the R plane.

It is rare in the underwater imaging, even when there is a reddish andhigh brightness area, that an area that causes the brightness saturationexists. Since the color blur area is narrow and the color blur intensityis small, the estimation parameter determination step S155 sets aparameter that narrows a color blur area and reduces the color blurintensity, and reducing a color blur estimation amount.

In a variety of scene settings, a proper color blur reduction objectplane and a parameter that is used to estimate a color blur area and thecolor blur intensity is set.

Third Embodiment

Next follows a description of a third embodiment of the presentinvention. The color image pickup apparatus 100″ of this embodimentincludes, as shown in FIG. 13, an imaging optical system 110, an imagesensor 120, an AD converter 130, a demosaic part 140, a color blurprocess determination part 150″, a color blur reducer 160, a sightcorrector 170, a compression part 180, and a recorder 190. Thoseelements which are the same as corresponding elements in the firstembodiment are designated by the same reference numerals. Thisembodiment is different from the first embodiment in a process of thecolor blur process determination part 150″.

FIG. 14 shows the color blur process determination part 150″ of thisembodiment. The color blur process determination part 150″ includes animage feature calculation step S151, a color blur reduction object planeselection step S152, a reference color plane selection step S153, anestimation method selection step S154, and an estimation parameterdeterminations step S155.

The image feature calculation step S151 uses a hue or a white balance(“WB”) of the entire image as a feature (or a characteristic) of theimage relating to the color blur. The hue may be said to be “a colortemperature” in different words, and the color temperature may be usedas the feature of the image.

The hue of the entire screen is greatly affected by the environmentallight, and the color blur is affected by a spectroscopic distribution ofthe light source. Therefore, due to the spectroscopic distribution ofthe environmental light, a color of a blur that is likely to occur in anarea having a large brightness difference can be estimated.

However, a color that blurs and is similar to the environmental light isless likely conspicuous and unlikely to be recognized as a clear colorblur; it is less necessity to consider the color blur as a color blurreduction object.

An image having a certain tint is unlikely to include only the samecolor as the hue of the entire image and, for example, a high brightnesslight source which has a color different from the hue of the entireimage is likely to exist in the image. In this case, it is conceivablethat a blur of a color different from the hue of the entire image occursaround the high brightness light source. The blur of the color differentfrom the tint of the entire image is likely to be perceived as a colorblur that causes a sense of discomfort. Therefore, it is reasonable thata color different from the hue of the entire image is set to a blurreduction object. A color plane that serves as a color blur reductionobject can be properly selected by calculating the hue of the entireimage.

In addition, depending upon the degree of the hue of the entire image,the color plane to be referred to as a reference needs to be changed. Anormal imaging optical system is designed with the G wavelength regionas a reference, and the chromatic aberrations from the R wavelength tothe B wavelength appears as a color blur although the chromaticaberrations differ according to the imaging optical systems. Since thiscolor blur occurs as a result of that the G wavelength is used forfocusing, the color plane to be referred to as a reference may usuallybe set to the G plane.

However, this is not the case where a wavelength other than the Gwavelength is in the on-focus state. In other words, when the entireimage has a few G wavelengths and another wavelength is in the on-focusstate, a blurred color is not the R or B plane, and a color blur cannotbe effectively reduced when the color plane to be referred to as areference remained to the G plane. Therefore, the color plane to bereferred to as a reference needs variable.

One image that has a few G wavelengths and can be focused with anotherwavelength is an image in which the hue of the entire image extremely isbiased to R and B. Therefore, a color plane to be referred to as areference can be properly selected by calculating the hue of the entireimage.

For example, assume imaging under the reddish environmental light, suchas in the evening or under the photoflood. When no color blur occurs inan area having a large brightness difference, the RGB planes are asshown in FIG. 15. However, imaging under the reddish environmental lightis likely to cause a red blur in an area having a large brightnessdifference as shown in FIG. 16, because the area has a large lightquantity of a long wavelength (R component) and a small light quantityof a short wavelength (B component).

However, as understood from FIGS. 15 and 16, the entire screen isreddish, no sudden color changes occur, and a slight red blur is lesslikely to stand out. The problem arises when there is a high brightnesslight source having a large light quantity amount of the shortwavelength (B component) in the reddish image. When the high brightnesslight source is saturated and imaged, as shown in FIG. 17, a blue bluris likely to occur around the brightness saturation area. When FIGS. 16and 17 are compared with each other, the blue is blurred and a colordifferent from the hue of the entire image occurs around the brightnesssaturation area, and is likely conspicuous. In addition, when the entireimage is reddish due to the influence of the environmental light, it isblended with a blue blur and becomes a conspicuous purple blur.

Thus, a conspicuous color blur can be identified by calculating the hueof the entire image. The image feature calculation step S151 calculatesthe hue of the entire image.

The color blur reduction object plane selection step S152 selects thecolor blur reduction object plane based on the hue calculated by theimage feature calculation step S151.

For a (first) image in which the entire image is reddish, the B plane isset to the plane as the color blur reduction object.

When the entire image is strongly reddish and the R wavelength is in theon-focus state, the G plane and the B plane blur as shown in FIG. 18,and a purple blur close to cyan is conspicuous. In that case, thereference color plane selection step S153 sets the color plane to bereferred to as a reference to the R plane. The color blur reductionobject plane selection step S152 sets the B plane and the G plane to thecolor blur reduction object plane.

Conversely, in a (second) image in which the entire screen is bluish, ablue blur is less likely conspicuous but a red blur turns into a purpleblur after blended with the hue of the entire image and then stands out.In that case, a proper plane setting is performed as in the aboveexample: For example, the color blur reduction object plane is set tothe R plane, and the color plane to be referred to as a reference is setto the B plane.

The estimation parameter determination step S155 determines a parameterthat is used to estimate a color blur area and a parameter thatdetermines the color blur intensity, according to the feature of theimage and the imaging condition obtained by the image featurecalculation step S151.

A stop value (an aperture value) is one that influences the color blurin the imaging condition. As the stop is opened, a depth of field(“DOF”) becomes small, an image blurs as a distance from a focusposition increases, and thus the color blur spread is likely toincrease. On the other hand, as the stop is closed, a DOF becomes large,a blur amount becomes small, and the color blur spread is likely toreduce. Therefore, a parameter that is used to estimate the color blurarea can be set according to the stop value at the imaging time.

A result of the color blur process determination part 150 is input tothe color blur reducer 160, which performs a proper color blur reductionprocess for each image based on the result.

One estimation method of the color blur amount performed by the colorblur reducer 160 detects a brightness saturation area on the premisethat a color blur occurs around the brightness saturation area, anddetermines a color blur amount according to a distance from thebrightness saturation area. This estimation method calculates a distancefrom the brightness saturation area shown in FIG. 19 and a distance fromthe brightness unsaturation area in a plane to be referred to as areference and a color blur reduction object plane. This method thencalculates a saturation radius from these calculated distances,estimates the light source intensity, and produces and estimates afunction model that damps according to a distance from the brightnesssaturation area.

In this case, a color blur is modeled by using a base value, a dampingparameter, a parameter indicative of the spread of a color blur, and thelike. Once a model matching the color blur is produced, the color blurcan be effectively reduced but a shift of the model causes aninsufficient reduction of a color blur or an unnatural color. Therefore,a parameter can be changed for each image, and a color blur amount isproperly estimated.

When this estimation method is used, the estimation parameter determinedby the estimation parameter determination step S155 becomes a basevalue, a damping parameter, and a parameter indicative of the color blurspread.

Fourth Embodiment

Next follows a description of a fourth embodiment. A color image pickupapparatus 100′″ of this embodiment includes, as shown in FIG. 20, animaging optical system 110, an image sensor 120, an AD converter 130, ademosaic part 140, a color blur process determination part 150′″, acolor blur reducer 160, a sight corrector 170, a compression part 180,and a recorder 190. Those elements which are the same as correspondingelements in the first embodiment are designated by the same referencenumerals. This embodiment is different from the first embodiment in aprocess of the color blur process determination part 150′″.

FIG. 21 shows the color blur process determination part 150′″. The colorblur process determination part 150′″ includes an image featurecalculation step S151, a color blur reduction object plane selectionstep S152, a reference color plane selection step S153, an estimationmethod selection step S154, and an estimation parameter determinationstep S155.

The image feature calculation step S151 calculates, as a feature (or acharacteristic) of an image, a brightness value of each color planearound a brightness saturation area in the image. When a color planearound the brightness saturation area has a higher brightness value thanthat of the G plane that is generally serves as a reference color plane,the color of the color plane is estimated to blur. The color blurreduction object plane selection step S152 selects the color plane as areduction object based on this result.

Since it is conceivable that a blurred color is different for eachbrightness saturation area in the image, a color plane that serves as acolor blur reduction object may be differently set for each area in theimage.

The reference color plane selection step S153 sets another plane havinga small brightness saturation area to a reference color plane based onthe result from the image feature calculation step S151 when thebrightness saturation area of the other color plane is smaller than thebrightness saturation area of the G plane around the brightnesssaturation area in the image. A phenomenon in which the brightnesssaturation area of a color plane other than the G plane is small islikely to occur when a wavelength other than the G wavelength is in theon-focus state. In addition, this phenomenon depends upon thespectroscopic distribution of the object in the brightness saturationarea in the image etc.

The estimation method selection step S154 selects an estimation methodof a color blur area and a color blur amount based on the brightnessvalue calculated by the image feature calculation step S151. This isbecause the estimation method includes a method of providing a colorblur reduction process only when there is a brightness saturation areaof a certain threshold or greater in the image, and a method ofproviding a color blur reduction process even when there is nobrightness saturation area in the image.

For example, a method of using a distance from the brightness saturationarea shown by the color blur reducer 160 of the third embodiment, and amethod of using a convolution process shown by the color reducer of thesecond embodiment are approaches of setting a color blur around thebrightness saturation area to a reduction object.

On the other hand, a method of using a brightness slope as in the colorblur reducer 160 of the first embodiment identifies a color blur by thebrightness slope, and sets to the reduction object a color blur thatoccurs in an area other than the brightness saturation area. In thesetting, when an image has a brightness saturation area of a certainthreshold or greater, an approach, such as a method of using a distancefrom the brightness saturation area and a method of using a convolutionprocess, is selected, and when there is no image that has a brightnesssaturation area of a certain threshold or greater, an approach, such asa method of using a brightness slope, is selected.

Alternatively, the estimation method selection step S154 may selectwhether a color blur reduction process is to be performed. For example,a switching action is provided such that the color blur reductionprocess can follow when an input image is determined to be a night viewimage in which a color blur stands out, based on the imaging conditionand the feature (or a characteristic) of the image obtained by the imagefeature calculation step S151, and no color blur reduction process canbe performed otherwise.

The estimation parameter determination step S155 determines a parameterthat is used for the estimation method selected by the estimation methodselection step S154, by using the imaging condition and the result ofthe image feature calculation step S151.

Fifth Embodiment

Next follows a description of the fifth embodiment of the presentinvention. The color image pickup apparatus 100 of this embodiment isthe same as the color image pickup apparatus 100 shown in FIG. 3 (or ofthe first embodiment). In this embodiment, the color blur processdetermination part 150 and the color blur reducer 160 provide the sameprocesses of the color blur process determination part 150 (or 150′,150″ or 150′″) and the color blur reducer 160 in the first to fourthembodiments.

This embodiment utilizes a type of an employed lens for an imagingcondition for the color blur reduction object plane selection step S152and the estimation parameter determination step S155 in the color blurprocess determination part 150 (see FIGS. 7, 10, 14 and 21) to determinea parameter. The chromatic aberration that causes a color blur differsaccording to a type of a lens, a chromatic aberration amount is known inthe design stage, and a color in which each type of lens is likely toblur and a color amount can be estimated. Therefore, this embodimentuses for the imaging condition a type of a lens employed for theimaging, and determines a color in which the lens is likely to blur anda blur amount, in that lens.

For example, in using a lens that has no chromatic aberration betweenthe R and G wavelengths and is likely to generate a blue blur due to theremaining chromatic aberration of the B wavelength, the color blurreduction object plane selection step S152 sets the B plane to thereduction object plane. In addition, the reference color plane selectionstep S153 in the process shown in FIGS. 14 and 21 sets the G plane to acolor plane to be referred to as a reference.

The estimation parameter determination step S155 sets a parameter of thecolor blur estimation intensity based on the remaining chromaticaberration amount.

Since the chromatic aberration that causes a color blur differsaccording to a zoom state, a zoom state may be used as the imagingcondition. Characteristics of a color of which the blur is likely tooccur at the wide angle side and a color of which the blur is likely tooccur at the telescopic side are previously known; a parameter that isused to estimate a color of a reduction object and a blur amount of thecolor can be set according to a type of a lens used for imaging and azoom state of the lens to be used.

For example, assume use of a lens that is likely to generate, due to aremaining chromatic aberration of the design, a blue blur as a distanceis close to the wide angle side, a red blur as a distance is close tothe telescopic side, and a few color blur in the intermediate zoomregion. The color blur reduction object plane selection step S152 setsthe R plane to the reduction object plane for an image obtained byimaging at the telescopic side, and sets the B plane to the reductionobject plane for an image obtained by imaging at the wide angle side. Inusing such a lens, the reference color plane selection step S153 can setthe G plane to a plane to be referred to as a reference.

The estimation parameter determination step S155 sets a parameter of thecolor blur estimation intensity based on the remaining chromaticaberration amount. In the above example with the lens to be used, aparameter is set such that the color blur area can spread in the imageobtained by imaging and the color blur intensity can increase as adistance approaches to the wide angle side and the telescopic side.

Since the chromatic aberration that causes a color blur differsaccording to an object distance used for focusing, the imaging conditionmay use an object distance (a distance from a lens to an object to befocused). The intensity of a color blur differs between a focusing stateonto a close object and a focusing state onto a distant object.

For example, assume use of a lens having a large color blur amount whenthe lens focuses onto a close object, a small color blur amount when thelens focuses onto a distant object. In that case, the estimationparameter determination step S155 sets a parameter that increases thecolor blur estimation intensity when the object distance is small, andreduces the color blur estimation intensity when the object distance islarge.

The color blur reduction method (image processing method) explained inthe above first to fifth embodiments may be used separately or incombination. In other words, the color blur process determination partmay change at least one of a color blur that serves as a reductionobject and an estimation method of an amount of the color blur accordingto at least one of a feature (or a characteristic) of an image and animaging condition. In other words, at least one of a color plane havingthe color blur that serves as the reduction object, a parameter used toestimate a color blur area, a parameter that is used to estimate thecolor blur intensity, a color plane to be referred to as a reference inthe color blur reduction process and a color blur estimation method maybe changed.

In addition, the above first to fifth embodiments discuss the colorimage pickup apparatus that includes all components from the imagingoptical system 110 to the recorder 190. However, as an apparatusseparate from the color image pickup apparatus, an image processingapparatus (such as a personal computer) having at least the color blurprocess determination part 150 (or 150′, 150″, or 150′″) and the colorblur reducer 160 may be formed. In this case, a color image taken by thecolor image pickup apparatus and stored in a recording medium, such as asemiconductor memory and a magnetic/optical disc, may be input into theimage processing apparatus. In addition, the color image pickupapparatus and the image processing apparatus may be connected to eachother via a cable, a radio LAN, etc. so as to send an image from thecolor image pickup apparatus to the image processing apparatus.

As described above, each embodiment can properly provide a color blurreduction process to images having various color blurs, effectivelyreduce the color blur of each image, and obtain a natural image withouta sense of discomfort.

In addition, this color blur reduction process can mitigate a limitationof a longitudinal chromatic aberration of the imaging optical system inthe image pickup apparatus, improve another aberrational correction, andreduce a size of the image pickup apparatus.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2007-298532, filed on Nov. 16, 2007, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing apparatus configured toperform a process to reduce a color blur in a color image, said imageprocessing apparatus comprising a color blur process determination partconfigured to change at least one of a color blur that serves as areduction object and an estimation method of an amount of the color bluraccording to at least one of a characteristic of an image and an imagingcondition, wherein the color blur process determination part changes atleast one of a color plane of the color blur as the reduction object, aparameter that is used to estimate an area of the color blur, aparameter that is used to estimate an intensity of the color blur, acolor plane that is referred to as a reference in the reduction processof the color blur, and an estimation method of the color blur, accordingto at least one of the characteristic of the image and the imagingcondition.
 2. An image processing apparatus according to claim 1,wherein the characteristic of the image includes at least one of abrightness of an entire image, a contrast of the entire image, a hue ofthe entire image, and a hue around a brightness saturation area in theimage.
 3. An image processing apparatus according to claim 2, whereinthe color blur process determination part sets, when a brightness of anentire image of a first image is lower than that of an entire image of asecond image, the color blur amount to be estimated to the first imagegreater than that to the second image.
 4. An image processing apparatusaccording to claim 2, wherein the color blur process determination partsets, when a contrast of an entire image of a first image is lower thanthat of an entire image of a second image, the color blur amount to beestimated to a first image greater than that to a second image.
 5. Animage processing apparatus according to claim 2, wherein the color blurprocess determination part sets to a blue plane a color plane having thecolor blur that serves as the reduction object when the image is reddishas a whole, and the color blur process determination part sets to a redplane a color plane having the color blur that serves as the reductionobject when the image is bluish as a whole.
 6. An image processingapparatus according to claim 1, wherein the imaging condition includesat least one of an imaging scene, a type of a lens used for imaging, azoom state of the lens, a stop value of the lens, and an object distancefrom the lens.
 7. An image processing apparatus according to claim 6,wherein the color process determination part sets, when a first image isan image of a night view imaging scene and a second image is an image ofan imaging scene other than the night view imaging scene, a color bluramount to be estimated to the first image greater than that to thesecond image.
 8. An image processing method configured to perform aprocess to reduce a color blur in a color image, said image processingmethod comprising a color blur process determination step of changing atleast one of a color blur that serves as a reduction object and anestimation method of an amount of the color blur according to at leastone of a characteristic of an image and an imaging condition, wherein,in the color blur process determination step, at least one of a colorplane of the color blur as the reduction object, a parameter that isused to estimate an area of the color blur, a parameter that is used toestimate an intensity of the color blur, a color plane that is referredto as a reference in the reduction process of the color blur, and anestimation method of the color blur is changed according to at least oneof the characteristic of the image and the imaging condition.
 9. Animage pickup apparatus comprising: an image pickup element configured topick up a color image; and an image processing apparatus configured toperform a process to reduce a color blur in the color image, said imagepickup apparatus comprising a color blur process determination partconfigured to change at least one of a color blur that serves as areduction object and an estimation method of an amount of the color bluraccording to at least one of a characteristic of an image and an imagingcondition, wherein the color blur process determination part changes atleast one of a color plane of the color blur as the reduction object, aparameter that is used to estimate an area of the color blur, aparameter that is used to estimate an intensity of the color blur, acolor plane that is referred to as a reference in the reduction processof the color blur, and an estimation method of the color blur, accordingto at least one of the characteristic of the image and the imagingcondition.
 10. An image processing apparatus configured to perform aprocess to reduce a color blur in a color image, said image processingapparatus comprising a color blur process determination part configuredto change at least one of a color blur that serves as a reduction objectand an estimation method of an amount of the color blur according to atleast one of a characteristic of an image and an imaging condition,wherein the characteristic of the image includes at least one of abrightness of an entire image, a contrast of the entire image, a hue ofthe entire image, and a hue around a brightness saturation area in theimage.
 11. An image processing apparatus according to claim 10, whereinthe color blur process determination part sets, when a brightness of anentire image of a first image is lower than that of an entire image of asecond image, the color blur amount to be estimated to the first imagegreater than that to the second image.
 12. An image processing apparatusaccording to claim 10, wherein the color blur process determination partsets, when a contrast of an entire image of a first image is lower thanthat of an entire image of a second image, the color blur amount to beestimated to a first image greater than that to a second image.
 13. Animage processing apparatus according to claim 10, wherein the color blurprocess determination part sets to a blue plane a color plane having thecolor blur that serves as the reduction object when the image is reddishas a whole, and the color blur process determination part sets to a redplane a color plane having the color blur that serves as the reductionobject when the image is bluish as a whole.
 14. An image processingapparatus according to claim 10, wherein the imaging condition includesat least one of an imaging scene, a type of a lens used for imaging, azoom state of the lens, a stop value of the lens, and an object distancefrom the lens.
 15. An image processing apparatus according to claim 14,wherein the color process determination part sets, when a first image isan image of a night view imaging scene and a second image is an image ofan imaging scene other than the night view imaging scene, a color bluramount to be estimated to the first image greater than that to thesecond image.
 16. An image processing method configured to perform aprocess to reduce a color blur in a color image, said image processingmethod comprising a color blur process determination step of changing atleast one of a color blur that serves as a reduction object and anestimation method of an amount of the color blur according to at leastone of a characteristic of an image and an imaging condition, whereinthe characteristic of the image includes at least one of a brightness ofan entire image, a contrast of the entire image, a hue of the entireimage, and a hue around a brightness saturation area in the image. 17.An image pickup apparatus comprising: an image pickup element configuredto pick up a color image; and an image processing apparatus configuredto perform a process to reduce a color blur in the color image, saidimage processing apparatus comprising a color blur process determinationpart configured to change at least one of a color blur that serves as areduction object and an estimation method of an amount of the color bluraccording to at least one of a characteristic of an image and an imagingcondition, wherein the characteristic of the image includes at least oneof a brightness of an entire image, a contrast of the entire image, ahue of the entire image, and a hue around a brightness saturation areain the image.